Development of a dual-modal information presentation of sequential relationship

Technologies for speech recognition and synthesis are becoming increasingly sophisticated and provide support for information processing via multi-modal interfaces. The benefit of delivering information across different sensory modalities is often backed by the independent nature of multi-modal information processing, which assumes that there will be no interference of modality between tasks and thus no degradation in performance (Cook et al. 1997). Research in cognitive psychology shows that visual and auditory perceptual processing is closely linked (Eimer 1999). However, problems related to memory and cognitive workload are found in applications with voice-based interface (Cook et al. 1997). For instance, mental integration of disparate information from different modality channels causes a heavy cognitive workload. As transient auditory information, speech presentation may impose a greater memory burden. Also, switching attention between modalities may be slow and have a high cognitive cost. The objective of this research is to design dual-modal interfaces to: (1) minimize the interference in information processing between the visual and auditory channels; and (2) improve the effectiveness of mental integration of information from different modalities. This study focuses on the dual-modal presentation of textual information that describes sequential relationships or chronological events. The proposed dual-modal presentation of the selected textual information describes such relationships among entities in diagrams and delivers the remaining information as voice messages. Users are expected to have superior comprehension performance and higher satisfaction while using the graph + voice presentation than using the pure textual presentation.